International Review of Research in Open and Distributed Learning 
Volume 17, Number 6 


December-2016 


Maintaining Momentum Toward Graduation: OER and 
the Course Throughput Rate 

John Hilton III 1 , Lane Fischer 2 , David Wiley 3 , and Linda Williams 4 

1 - 2 ’ 3 Brigham Young University, 3 Lumen Learning, 4 Tidewater Community College 


Abstract 

Open Educational Resources (OER) have the potential to replace traditional textbooks in higher 
education. Previous studies indicate that use of OER results in high student and faculty satisfaction, 
lower costs, and similar or better educational outcomes. In this case study, we compared students using 
traditional textbooks with those using OER at Tidewater Community College to compare their 
performance on what we call course throughput rates, which is an aggregate of three variables - drop 
rates, withdrawal rates, and C or better rates. Two self-selecting cohorts were compared over four 
semesters, with statistically significant results. The study found that, subject to the limitations discussed, 
students who use OER perform significantly better on the course throughput rate than their peers who 
use traditional textbooks, in both face-to-face and online courses that use OER. This suggests that OER 
are a promising avenue for reducing the costs of higher education while increasing academic success. 
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Introduction 

The high cost of textbooks represents a significant challenge in higher education in the United States. A 
survey of 22,129 post-secondary students in Florida found that 64% of students reported not purchasing a 
required textbook because of its high cost (Florida Virtual Campus, 2012). In the same study, nearly half 
of students reported that the cost of textbooks caused them to take fewer courses, and one-third stated 
that they had earned a poor grade in a subject because they could not afford to buy the textbook. 

One solution to the high cost of textbooks is found in a replacement for textbooks called Open 
Educational Resources (OER). The term Open Educational Resources comes from the 2002 UNESCO 
Forum on the Impact of Open Courseware for Higher Education in Developing Countries, which defined 
OER as “The open provision of educational resources, enabled by information and communication 
technologies, for consultation, use and adaptation by a community of users for non-commercial purposes” 
(UNESCO, 2002, p. 24). In essence, OER are free educational materials that are licensed in such a way so 
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as to grant legal permissions to reuse, remix, and redistribute them. Wiley, Bliss, and McEwen (2014) 
chronicle many aspects of the growth and development of OER since its inception. 

To date, a wide variety of high-quality OER have been created, although there are varying quality of OER 
and they are not all a natural substitute for conventional paid material. Nevertheless, many OER are 
sufficiently robust so as to be useful in replacing traditional textbooks. These OER typically have Creative 
Commons licenses that provide the legal permissions necessary to free share, modify, and reuse them 
(Bissell, 2009; D’Antoni, 2009; Hewlett, 2013). The Minnesota Open Textbook Library 
(open.umn.edu/opentextbooks/) provides a repository of open textbooks and also hosts faculty reviews of 
these materials. Despite the widespread intuition that freely available educational materials must be less 
effective or of lower quality than expensive, published materials, emerging research demonstrates 
otherwise. 

Although available OER has grown dramatically in the past decade, Allen and Seaman (2014) found that 
across a nationally representative survey of 2,144 college faculty members in the United States, only 34% 
of respondents were aware of OER. However, several studies have shown faculty who are familiar with 
OER tend to have positive perceptions of it. Pitt (2015) demonstrated that faculty who utilize OER viewed 
it positively. She surveyed 127 teachers who adopted an open textbook and found that approximately 75% 
of faculty members said that after using open textbooks as their primary teaching tool, they would 
continue to use open textbooks in the future. Similarly, Bliss, Robinson, Hilton, and Wiley (2013) 
examined OER adoption at several colleges and surveyed approximately sixty faculty members regarding 
their experiences using OER. They found that 55% of teachers who had used OER said that the OER were 
of the same quality as the materials that were previously used, and 35% felt that they were better, with the 
remaining 15% feeling that they were worse. A recent study of faculty perceptions of OER at British 
Columbia post-secondary institutions produced similar results (Jhangiani, Pitt, Hendricks, Key, & 
Lalonde, 2016). Similar positive findings have also been demonstrated at the secondaiy level (Kimmons, 
2015). 

Students also tend to be very positive about their use of OER. Feldstein, Martin, Hudson, Warren, Hilton, 
and Wiley (2012) surveyed 1,393 students at Virginia State University who utilized OER as a substitute for 
traditional textbooks. Out of the 315 students who responded, 95% strongly agreed or agreed that OER 
were “easy to use,” and 78% felt OER “provided access to more up-to-date material than is available in my 
print textbooks” (para. 29). Two-thirds of students either agreed or strongly agreed that the digital OER 
were more useful than traditional textbooks and stated they preferred OER to traditional textbooks. One 
reason students support OER relates to the cost savings, significant in many instances. Huggins and 
Smith (2015) wrote that the launch of an OER initiative at Kaplan University had led to cost-savings of 
more than one million dollars over a one-year period of time. 

While cost-savings are important to some educators, the more vital issue relates to student learning. 
Pawlyshyn, Braddlee, Casper, and Miller (2013) reported on the adoption of OER at Mercy College. The 
researchers compared the pass rates of students who used OER in the Fall of 2012 with those who had 
used traditional materials in the Fall of 2011. The pass rates of math courses increased from 63.6% in Fall 
2011 to 68.9% in Fall 2012. The contrast between the Spring of 2011 (no OER, pass rate of 48.4%) and the 
Spring of 2013 (OER, pass rate of 60.2%) is even more significant. However, it must be pointed out that 
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the change in textbooks coincided with a change in pedagogy, which may have been the decisive factor in 
the growth in student learning. 

Other studies have not shown such dramatic results when OER were adopted. Allen et al. (2015) 
examined the use of a collection of OER called ChemWiki in a chemistry class taught at the University of 
California-Davis. Students in back-to-back hours were taught by the same instructor with the same 
teaching assistants. In one section, 478 students used ChemWiki as the primary learning resource, while 
the other class (of 448 students) used a traditional textbook. Pretests indicated no significance differences 
between the two groups. All students took the same midterm and final exam, and there were no 
significant differences between the overall results of the two groups. A review of several additional studies 
by Hilton (2016) indicates that generally, students who use OER tend to do as well or better than their 
peers using traditional textbooks in terms of course completion and passing rates. 

Thus, the literature seems to indicate that open textbooks are connected with high student and faculty 
satisfaction, lower costs, and similar or better educational outcomes. However, it should be noted that the 
literature to date is relatively sparse. One aspect that has yet to be studied is the relationship between the 
students dropping courses and courses that utilize open textbooks. One theory put forth by Wiley, 
Williams, DeMarte and Hilton (2016) is that if students who use OER drop classes at lower rates than 
their peers who use traditional textbooks, then institutions of higher education stand to gain financially 
through OER adoption, as they will retain tuition money that they would otherwise need to refund. These 
researchers studied drop rates at Tidewater Community College in the Fall 2013 and Spring 2014 
semesters. They found that there was a small but statistically significant difference in drop rates between 
courses using OER versus those that utilized traditional textbooks. This study was limited by the small 
amount of data available in the initial pilot semesters, as well as in that the data for both online and face- 
to-face sections were aggregated and not reported separately. 

Another issue connected to drop rates is the overall rate of student success, which we refer to in this paper 
as “course throughput rate.” There are multiple filter points at which a student might not successfully 
“survive” a class. A student could drop the class before the add/drop deadline, or might withdraw from 
the class, or could persist in the class but not pass it. The purpose of the present study is to expand on the 
research done by Wiley et al. (2016) and examine course throughput rates across the four semesters of the 
pilot program at Tidewater Community College. Specifically, the research questions that guide the present 
study are as follows: 

1. What was the difference (if any) in the drop rates between students taking courses using OER 
versus those using traditional textbooks for both online and face-to-face classes? 

2. What was the difference (if any) in the withdrawal rates between students taking courses using 
OER versus those using traditional textbooks for both online and face-to-face classes? 

3. What was the difference (if any) in the proportion of students getting a C grade or better in the 
courses between students taking courses using OER versus those using traditional textbooks for 
both online and face-to-face classes? 
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4. What is the cumulative impact of these three effects (if any), which we call the course throughput 
rate? 


Context 

Founded in 1968 as a part of the Virginia Community College System, Tidewater Community College 
(TCC) has multiple campuses in Virginia. TCC is the largest provider of higher education and workforce 
services in Hampton Roads, Virginia, enrolling almost 40,000 students annually — the second largest 
undergraduate student body in the Commonwealth of Virginia. TCC is 14th largest public two-year 
community college in the U.S., and the second largest provider of undergraduate public education in 
Virginia. It also has the largest undergraduate African American enrollment in Virginia higher education, 
and is the seventh largest associate degree producer among two-year colleges for African American 
students in the nation. TCC is the 16th largest associate degree producer in the U.S. among two-year 
institutions and offers 12 nationally accredited degree programs. The student body is diverse and is 
comprised of 45% White, 34% African American, and 11% other minorities. Of the students attending TCC, 
either full or part-time, 56% receive financial aid (TCC Fact Book, 2015). 

In January 2013, Tidewater Community College began the process of becoming the first college in the U.S. 
to create an Associate of Science degree based entirely on openly licensed content. This program is 
referred to as a “Z Degree,” referring to a degree with “Zero” textbook costs. The Z Degree is made up of a 
series of “Z Courses,” which are courses with “Zero” textbook costs. The goals of the Z Degree are 
threefold: 1) to improve student success, 2) to increase instructor effectiveness, and 3) to save students 
money. Courses were stripped down to the learning outcomes and rebuilt using openly licensed content, 
reviewed and selected by faculty based on its ability to facilitate student achievement of the objectives. 

Acknowledging the tendency for course design to be based upon anecdotal feedback or “gut feel,” the 
courses that make up the Z Degree were designed in a systematic data-driven process that closely 
examined existing curriculum frameworks and sought ways to increase their efficiency and 
effectiveness. Essential to this process was the implementation of an outcomes based backwards 
design process. The argument for backwards curriculum design and laser- focused alignment of course 
materials has been made after realizing that this process gives educators the ability to control curriculum, 
use current information, and most importantly, foster student success. The combined efforts of a 13- 
member faculty team, college staff, and administration culminated on August 22, 2013 when more than 
420 students enrolled in the first 16 “Z Courses.” Additional courses were launched in Spring 2014, 
completing the path to the degree for TCC’s business administration students. The courses that comprise 
the Z Degree are designated as “Z courses” within the registration system that students use to enroll in 
classes. These Z courses are delivered online, face-to-face, and in hybrid/blended formats and are taught 
by both full-time and adjunct faculty. Enrollment in Z courses is open to all TCC students, regardless of 
their field of study. They use the same process for enrollment as the other courses at TCC and are filled on 
a first-come, first-served basis. 


Method 
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Data for this case study were drawn from the Tidewater Community College institutional research 
database in Fall 2013, Spring 2014, Fall 2014, and Spring 2015. Data included the drop rates, withdrawal 
rates, and final grades in courses with non-Z and Z sections in the same semester. Most Tidewater classes 
are 16-week courses; however, 8 and 12-week versions of some classes are available. We obtained drop 
rate data for only the 16-week courses, and withdrawal and grade data for all courses, leading to a 
difference in the total n population between these figures. 

Outcomes were analyzed separately according to their modality: face-to-face or online/hybrid. Data were 
aggregated across four semesters: Fall 2013, Spring 2014, Fall 2014, and Spring 2105. Non-Z sections 
(sections requiring commercial publisher materials) were labeled control, while Z-sections (OER) were 
considered treatment. There were 67 courses taught across the four semesters that had both treatment 
and control sections. Because of the much greater availability of traditional (Non-Z) courses, there were 
notably more students enrolled in control sections than treatment sections. Because of the discrepancy in 
sample sizes, unequal variances were assumed in each statistical test. For the course throughput rate we 
computed Cohen’s d using a logit method. 


Results 

Table 1 summarizes the results of our analyses: 

Table 1 

Differences Between Non-Z and Z Courses 


Outcome 

Face to Face 

Online/Hybrid 

Drop Rate 

Control n = 30,013 

Treatment n = 1,175 

Control % Drop = 2.3 

Treatment % Drop = 1.8 

Z = 1.29 

P = 0.19 

Control n = 3,333 

Treatment n = 703 

Control % Drop = 4.0 

Treatment % Drop = 1.4 

Z = 4.66 
p < 0.001 

Withdrawal Rate 

Control n = 36,223 

Treatment n = 1,151 

Control % Withdrawal = 9.9 
Treatment % Withdrawal = 8.1 

Z = 2.07 
p = 0.04 

Control n = 7,000 

Treatment n = 863 

Control % Withdrawal = 13.7 

Treatment % Withdrawal = 13.1 

Z= 0.52 
p = 0.60 

Grade > C 

Control n = 36,223 

Treatment n = 1,151 

Control % > C = 68.0 

Treatment % > C = 73.7 

Z=- 4.29 
p < 0.001 

Control n = 7,000 

Treatment n = 863 

Control % > C = 65.5 

Treatment % > C = 69.8 

Z= -2.58 
p = 0.009 
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Course Control n = 36,223 Control n = 7,000 

Treatment n = 1,151 Treatment n = 863 

Throughput Control % Success = 59.8 Control % Success = 54.2 

Treatment % Success = 66.4 Treatment % Success = 59.8 

Rate Z = -4.66 Z=- 3.13 

_ p < O.OOl _ p = 0.002 _ 

Note: Calculations: Control group proportions minus treatment group proportions 

The first research question we asked was whether there was a significant difference between the control 
and treatment groups in terms of the proportion of students that dropped the course before the deadline 
to receive a tuition refund. In face-to-face courses, 2.3% of the control subjects dropped, while 1.8% of 
treatment subjects dropped their courses. A two sample Z-test of differences in proportions rendered a 
value of 1.29, which was not statistically significant (p = 0.19). In online/hybrid courses, 4.0% of the 
control subjects dropped, while 1.4% of treatment subjects dropped their courses. A two sample Z-test of 
differences in proportions rendered a value of 4.66, which was significant (p < 0.001). 

The second research question asked whether there were significant differences between the control and 
treatment groups in terms of the proportion of students that withdrew after the drop deadline and did not 
receive a tuition refund. In face-to-face courses, 9.9% of the control subjects withdrew, while 8.1% of 
treatment subjects withdrew. A two sample Z-test of differences in proportions rendered a value of 2.07, 
which was significant (p = 0.04). In online/hybrid courses, 13.7% of control subjects withdrew, while 13.1% 
of the treatment subjects withdrew. A two sample Z-test of differences in proportions rendered a value of 
0.52, which was not significant (p = 0.60). 

The third research question asked whether there was a significant difference between the control and 
treatment groups in the proportion of students that achieved a C grade or better. In face-to-face courses, 
68.0% of control subjects received a C grade or better, while 73.7% of the treatment subjects received a C 
grade or better. A two sample Z-test of differences in proportions rendered a value of -4.29, which was 
significant (p <.001). In online/hybrid courses, 65.5% of control subjects received a C grade or better, 
while 69.8% of the treatment subjects received a C grade or better. A two sample Z-test of differences in 
proportions rendered a value of -2.58, which was significant (p = 0.009). 

The fourth research question combined the drop, withdrawal, and C or better rates into a single metric we 
call the “course throughput rate” to estimate the differences between the groups in their overall success 
rate from registration to final grade. We used the following calculations to determine the course 
throughput rate: (total students registered on day 1) x (l-drop rate) x (l-withdrawal rate) x (percent 
passing with a C or better) / (total students registered on day 1) = course throughput rate. In the face-to- 
face courses, 59.8% of students in non-Z courses made it through the successive hurdles of drop, 
withdrawal, and passing the class, compared with 66.4% of students in the Z courses, for a difference of 
6.6%. A two sample Z-test of differences in proportions rendered a value of -4.66, which was significant (p 
<.001). Cohen’s d = 0.15, a positive but small effect. In the hybrid/online courses, 54.2% of students who 
started in non-z courses successfully made it through the course with a C or better, compared with 59.8% 
of students in the Z courses, for a difference of 5.6%. A two sample Z-test of differences in proportions 
rendered a value of -3.13, which was significant (p = .002). Cohen’s d = 0.12, which, similar to the face-to- 
face courses, represents a positive but small effect. 


23 



Helping More Students to Graduate on Time: Open Educational Resources as a Lever 
Hilton, Fischer, Wiley, and Williams 


Discussion 

The results of this study indicate that students do no worse academically when they enroll in course 
sections that do not require them to purchase commercial textbooks. In fact, students in face-to-face Z 
courses were significantly less likely to withdraw from a course and more likely to receive a C or higher in 
the course than their peers who took non-Z courses. Overall, these students enrolled in face-to-face Z 
courses were almost 7% more likely to succeed than those who took non-Z courses. Similarly, students in 
hybrid/online Z courses were significantly less likely to drop out of a course and more likely to receive a C 
or higher in the course than their peers who took non-Z distance courses. The success rate of students in 
the distance Z courses was nearly 6% higher than those in the distance non-Z courses. Whether in face-to- 
face or online courses, these improved student success rates translate into more students moving forward 
toward graduation without repeating courses. 

The design of this study cannot establish causation, and one must be careful in making statements about 
results, particularly given the limitations described in the following section. At the same time, the results 
of this study are roughly in line with similar previous studies that have shown that students enrolled in 
courses that use OER perform as well or better than their peers who use traditional textbooks (Hilton, 
2016). Thus, this study adds to the body of research suggesting that OER are a promising avenue for 
reducing the costs of higher education without compromising academic success. It may be that increasing 
student access to learning materials is connected with their increased academic success. 

A unique contribution of the present study is that it examines the changes to the course throughput rate of 
students whose faculty assign OER. While individually the differences in drop, withdrawal, and passing 
rates are important, it is also useful to examine their collective influence. In the present study, students in 
both face-to-face and distance Z courses performed higher in two of these three categories, which led to a 
larger cumulative effect in student success. Not only did students in the face-to-face courses pass the class 
at a higher rate than those enrolled in the non-Z courses, but there was also a higher proportion of 
students who did not withdraw. While we might hypothesize that students who withdrew from a course 
would have done worse than those who do not, in this case, even though the face-to-face Z classes had a 
higher proportion of students who did not withdraw, they still performed better than their peers. Similarly, 
we might suspect that students who drop a course are less likely to pass the class than students who 
remained. Thus, we would hypothesize that in the hybrid/online Z sections, the students who remained 
would perform worse since their peers because fewer of them dropped out initially. However, students 
passed the class at higher rates, even though the proportion of those who had dropped the class was 
smaller. 


Limitations 

While the results of this study do suggest significant value stemming from courses offered with zero 
textbook costs, there are a number of limitations which must be addressed, particularly since we are only 
presenting a case study of a pilot program. One of these limitations is the large disparity in numbers 
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between the enrollment numbers in the non-Z and Z sections of courses. The Z-degree program is still in 
its infancy, and the total number of Z courses offered is small relative to the non-Z courses. A related 
weakness is that OER adoption or non-adoption is perfectly confounded with faculty in this analysis. It is 
possible that the observed differences in course throughput rates are more attributable to differences in 
faculty than differences in the price and licensing of the required instructional materials they assigned. 
Given that the teachers of Z-degree courses in these four pilot semesters were largely those who were 
involved in the creation of the courses (and may have been selected for this role because of their teaching 
abilities), there is a possibility that selection bias plays an important factor in the present study. 

An additional limitation of this study is that we do not have data indicating that the student groups in the 
two sets of courses are equivalent. It is possible that certain types of students specifically seek out Z 
courses, and that these students tend to do better academically. At the same time, we do not have any 
evidence suggesting this is the case; indeed, it is equally possible that the students who seek out Z courses 
would generally tend to do worse academically, thus understating the results of this study. This limitation 
could be addressed in future studies by focusing on a few specific courses where pre-tests could help 
assess the initial equivalency of the two groups. 


Conclusion 

We believe that the course throughput rate, which provides the combined effect of drops, withdrawals, 
and final grades, is a critical student success metric that merits additional study and perhaps expansion. 
Improving any of its component measures is only potentially helpful - students who do not drop a course 
may still withdraw and students who do not withdraw from a course may still fail. Consequently, the most 
powerful educational interventions may be those that lead to increases in the overall course throughput 
rate, which could in turn lead to higher graduation rates. While additional methodological rigor will be 
required in future studies before OER can be said to cause improvements in course throughput rates, the 
current study demonstrates that OER adoption by faculty is associated with improvements in course 
throughput rates. This finding merits further exploration. 
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